Active Lens Defogging For Digital Imaging Systems

ABSTRACT

A lightfield otoscope includes a housing with a tip configured to receive a disposable speculum. The otoscope also includes a microlens array, a sensor array and an optical train contained within the housing. The optical train includes an objective lens and a relay lens. The objective lens is positioned at least partially within the tip. The relay lens is used to relay an image plane of the objective lens to the microlens array and to relay a pupil plane of the objective lens to the sensor array. An active heating element is also contained within the housing and positioned to heat the front surface, thereby reducing fogging and/or condensation on the front surface.

BACKGROUND 1. Technical Field

This disclosure relates generally to lens defogging of digital imagingsystems, including lightfield otoscopes.

2. Description of Related Art

Digital camera systems are sometimes required to work in extremeenvironmental conditions. These environmental conditions include hightemperatures and relative humidity, such as in a body cavity, wheretemperatures can reach 45° C. (113° F.). The relative humidity is afunction of environment, and in vivo the relative humidity can reach100%.

A lightfield otoscope is a clinical imaging system that collects 3Dinformation about the patient's ear canal. The lightfield otoscope'sprimary imaging target is the tympanic membrane. The otoscope isrequired to work in a high-temperature and high-humidity environment. Ifa patient has a high fever in a wet season, common environmental factorsassociated with disease states, fog and condensation may form on thefront surface of the lightfield otoscope during the exam. The fog willprevent capture of good quality images of the ear drum.

SUMMARY

In one aspect, a lightfield otoscope includes a housing with a tipconfigured to receive a disposable speculum. The otoscope also includesa microlens array, a sensor array and an optical train contained withinthe housing. The optical train includes an objective lens and a relaylens. The objective lens is positioned at least partially within thetip. The relay lens is used to relay an image plane of the objectivelens to the microlens array and to relay a pupil plane of the objectivelens to the sensor array. An active heating element is also containedwithin the housing and positioned to heat the front surface, therebyreducing fogging and/or condensation on the front surface.

Other aspects include components, devices, systems, improvements,methods, processes, applications, computer readable mediums, and othertechnologies related to any of the above.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

Embodiments of the disclosure have other advantages and features whichwill be more readily apparent from the following detailed descriptionand the appended claims, when taken in conjunction with the accompanyingdrawings, in which:

FIGS. 1A-1B illustrate the effect of fogging on images, according to anembodiment.

FIG. 2 illustrates a lightfield digital otoscope system, according to anembodiment.

FIG. 3A is a cross-section of a lightfield otoscope, according to anembodiment.

FIG. 3B is a diagram of the optical train from FIG. 3A, according to anembodiment.

FIG. 4 is a diagram of a lightfield otoscope with an anti-foggingheating element, according to an embodiment.

FIG. 5 is a sequence of images of an ear canal phantom taken over timein the presence of fogging, according to an embodiment.

FIG. 6A shows the power spectra for red, green and blue channels takenover time in the presence of fogging, according to an embodiment.

FIG. 6B shows the power as a function of time for one specificfrequency, according to an embodiment.

FIG. 7 is a sequence of images of an ear canal phantom taken over timewhile using an active heater element, according to an embodiment.

FIG. 8A shows the power spectra for red, green and blue channels takenover time while using an active heater element, according to anembodiment.

FIG. 8B shows the power as a function of time for one specificfrequency, according to an embodiment.

FIG. 9A plots different trials with the heater element off, according toan embodiment.

FIG. 9B plots different trials with the heater element on, according toan embodiment.

The figures depict various embodiments for purposes of illustrationonly. One skilled in the art will readily recognize from the followingdiscussion that alternative embodiments of the structures and methodsillustrated herein may be employed without departing from the principlesdescribed herein.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The figures and the following description relate to preferredembodiments by way of illustration only. It should be noted that fromthe following discussion, alternative embodiments of the structures andmethods disclosed herein will be readily recognized as viablealternatives that may be employed without departing from the principlesof what is claimed.

FIGS. 1A-1B illustrate the effect of fogging on images, according to anembodiment. FIG. 1A is an image of a healthy tympanic membrane collectedwith a lightfield otoscope. FIG. 1B is an image of the same patienttympanic membrane after the front window of the lightfield otoscope hasfogged. Between the clear and fogged images, the high frequency spatialfeatures are no longer in focus and the color quality of the image isdeteriorating. Spatial features identifiable in FIG. 1A are notidentifiable in FIG. 1B. This is detrimental because conditions such asacute otitis media (AOM) and otitis media with effusion (OME) can bediagnosed based on shape, color and/or features of the tympanicmembrane.

FIG. 2 illustrates use of a lightfield otoscope 200 to image the eardrum250 of a patient. In this example, the otoscope 200 is handheld andincludes a main housing 210 and a handle 260. A disposable speculum 205is attachable to the tip 212 of the housing 210. In this example, theoutput of the otoscope 200 is transmitted via electronics and interface270 to a computer system 280, which processes the captured lightfieldimages and displays the desired results.

The main housing 210 contains the optical train, which is shown in moredetail in FIGS. 3A and 3B. The optical train includes an otoscopeobjective lens 310 and a relay lens 320A-B. Here, the term “lens” ismeant to include groups of optical elements. For example, the objectivelens 310 includes multiple optical elements, and the relay lens 320includes two groups of optical elements. The otoscope also includes amicrolens array 330 followed by a sensor array 340. The objective lens310 forms an image of the object of interest (i.e., the tympanicmembrane). The relay lens 320 relays this image to the microlens array330. That is, the microlens array 330 is positioned at a conjugate I2 ofthe image plane I1 of the objective lens 310. The sensor array 340 ispositioned at a conjugate P3 of the pupil plane P1 of the objective lens310, typically at the focal plane of the microlens array 330. The handle260 includes an illumination source 360 for the otoscope 200.

The optical train forms two overlapping imaging subsystems. One imagingsubsystem images the object onto image plane I1, which is then relayedto the microlens array 330 located at conjugate plane I2. The otherimaging subsystem images the pupil plane P1 onto the sensor array 340located at conjugate plane P3. The system in its entirety formsspatially multiplexed and interleaved optical images at the sensor array340. For convenience, the image captured by the sensor array 340 will bereferred to as a lightfield image. For convenience, the locations ofimages, apertures and their optical conjugates will be referred to asplanes (e.g., image plane, pupil plane), but it should be understoodthat the surface does not have to be perfectly planar.

The lightfield image has the following structure. The object is imagedonto I2. In conventional imaging, a sensor array at I2 would capturethis image. In the lightfield otoscope, a microlens array is located atI2. Each microlens is illuminated by light from a certain region of theobject. For example, the center microlens is illuminated by light fromthe center region of the object. The lightfield image has a structure ofsuperpixels corresponding to the microlenses. Each superpixel captureslight from a certain region of the object. The superpixels contain manyindividual subpixels, which typically correspond to individual sensorsin the sensor array. Each subpixel within a superpixel captures lightfrom the same region of the object, but at different propagation angles.

In other words, the object generates a four-dimensional light fieldL(x,y,u,v), where L is the amplitude, intensity or other measure of aray originating from spatial location (x,y) propagating in direction(u,v). Each subpixel in the lightfield image captures light from acertain volume of the four-dimensional light field. The subpixels aresampling the four-dimensional light field. The shape or boundary of suchvolume is determined by the characteristics of the lightfield imagingsystem.

In certain lightfield imaging system designs, the sample volumes arehyperrectangles. That is, every subpixel within a superpixel captureslight from the same rectangular (x,y) region associated with thesuperpixel, and each subpixel within the superpixel captures light froma different rectangular (u,v) region. However, this is not always thecase. For convenience, the superpixels will be described as capturinglight from a certain region of the object (even though subpixels withinthat superpixel may capture light from slightly different regions), andthe subpixels will be described as capturing light from a certain rangeof propagation directions (even though the range may be different fordifferent subpixels, or even for different (x, y) points captured by thesame subpixel). Regardless of the details, the sensor array 340 capturesa lightfield image, which maps (x,y) spatial locations and (u,v)propagation directions to subpixels. This is in contrast to aconventional image, which maps (x,y) spatial locations to pixels butloses information about the (u,v) propagation directions.

Optionally, a filter module can be positioned at the pupil plane (or oneof its conjugates P2 in this example). The filter module can contain anumber of spatially multiplexed filter cells, which allows filtering ofthe lightfield image.

Because the lightfield image contains information about thefour-dimensional light field produced by the object, the processingmodule 280 can be used to perform different types of analysis, such asdepth estimation, three-dimensional reconstruction, syntheticrefocusing, extending the depth of focus, spectral analysis and othertypes of multi-view analysis.

The characteristics of a lightfield imaging system can be usedadvantageously in otoscopes to image the interior of the ear. By using alightfield imaging system, three-dimensional (3D) shapes, translucencyand/or color information can be captured and extracted.

For example, the lightfield otoscope may be operable in a depth imagingmode. In the depth imaging mode, the lightfield image captured by thesensor array is processed to provide a three-dimensional depth image ofan inside of an ear. Alternately or additionally, a lightfield otoscopeis operable in a spectral imaging mode. In the spectral imaging mode,lightfield data captured by the sensor array is processed to provide twoor more different spectral images of an inside of an ear. Disparity ordepth maps can also be determined. The lightfield otoscope may beswitchable between the depth imaging mode and the spectral imaging modeor operate in both.

The lightfield data can be processed to produce enhanced imagery of theear interior. Data based on the enhanced imagery can then be used toassist a person in making a medical diagnosis. This diagnostic datacould be the enhanced imagery itself or it could involve furtherprocessing of the enhanced imagery.

Enhanced imagery of the tympanic membrane is a good example. Alightfield otoscope can simultaneously capture depth and spectralinformation about the tympanic membrane. A depth map of the tympanicmembrane can produce information regarding its shape—whether it isbulging or retracting, and the estimated curvature. Spectral informationcan include an amber or yellow image, which is especially useful todiagnose conditions of the tympanic membrane.

For example, Table 1 lists some features distinguishing the conditionsof acute otitis media (AOM), otitis media with effusion (OME), andotitis media with no effusion. As can be seen from Table 1, the threeconditions of the ear are different and they can be distinguished fromone another based on one or more of the following features: color,position (e.g., 3D shape), and translucency. In order to make correctdiagnosis of the ear condition, otoscopic images capturing accurateinformation about color, 3D shape and translucency of an inside of anear (e.g., a tympanic membrane in an ear canal) are desirable. These canall be captured simultaneously by a lightfield otoscope.

TABLE 1 Otoscopic findings associated with clinical diagnosticcategories on TM images AOM OME NOE Color White, pale yellow, White,amber, gray, Gray, pink markedly red blue Position Distinctly full,Neutral, retracted Neutral, bulging retracted Translucency OpacifiedOpacified, semi- Translucent opacified

Lightfield data also includes multiple views of the same image. Thisallows the user to refocus to different depths in the image and to viewthe same image from different viewpoints. For example, the effect ofoccluding objects may be reduced by taking advantage of multiviews. Thiscould be accomplished by refocusing. Alternately, it could beaccomplished by segmenting the lightfield (multiple views) into depthlayers.

However, as mentioned previously, the ear canal can be hot and humid.Fogging and condensation on the front surface of the lightfield otoscopecan degrade the quality of the captured images. FIG. 4 is a diagram of alightfield otoscope with an anti-fogging heating element, according toan embodiment. FIG. 4 shows the tip 212 of the otoscope housing, with aspeculum 205 mounted on the tip. The objective 310 and relay 320Aelements of the optical train are also shown. The objective lens 310 isheld in a lens barrel 415, which is made of a thermally conductivematerial. The housing contains a nichrome wire 440, which is positionedto heat the front surface 410 of the optical train, thereby reducingfogging and condensation. A controller, for example implemented onprinted circuit board 444, controls current to the nichrome wire 440. Asthe wire generates heat, the heat is conducted through the lens barrel415 to the front surface 410. An insulating layer, such as thedisposable speculum 205, shields the patient's ear drum from heating bythe wire.

In more detail, the two ends 442A,B of the wire 440 are located in thebase of the tip (i.e., the part that is away from the ear). The wire 440runs along the lens barrel from the base to the front surface and back,so that a mid-section of the wire is located near the front surface. Thetwo ends 442 are connected to the controller, for example electroniccircuitry that regulates the current running through the wire 440.Because this is an otoscope application, the controller controls theheating to avoid any injury to the patient. In addition, the controllerhas an operating range that includes at least 36-40 degrees Celsius,since those are typical conditions to be encountered.

Heating can be manually controlled. For example, the otoscope might havea button that is pressed by the operator to activate heating.Alternatively, the active heating element may run continuously duringoperation of the otoscope to maintain the front surface 410 at a hightemperature to avoid fogging, even without manual instructions from theoperator. In some cases, the controller may control the heating elementbased on environmental conditions in the vicinity of the front surface410, for example the temperature or humidity in this vicinity.

In yet another approach, heating is controlled based on the quality ofthe captured images. FIG. 5 is a sequence of images of an ear canalphantom taken over time in the presence of fogging when the activeheater is off. Condensation on the front surface 410 fogs the image. Thephotos are taken approximately every six seconds. The ear canal phantomis 15 mm long and 10 mm in diameter. At the end of the ear canal is aset of pie-shaped wedges arranged in the shape of a spiral staircase.Each step is 300 μm tall. This is a phantom for the ear drum. The designis based on FEM analysis and physiological studies of ear anatomy. Theexperimental setup is also capable of holding a temperature and humidityprobe to measure the environmental conditions in the phantom ear canal.This ear canal phantom was set to 40.2° C. and 104% humidity. Alightfield otoscope was used to produce the images in FIG. 5 over aperiod of 24 seconds. The images clearly show fog forming over theimage.

FIGS. 6A and 6B show the corresponding power spectra. FIG. 6A (top rowof three graphs) shows the power spectra. The left graph is for the redcomponent, the middle graph is for the green component and the rightgraph is for the blue component. Each graph shows a series of curves.Each curve is for a different time. The top curve is for the earliesttime (least fogging) and the bottom curve is for the latest time (mostfogging). The x-axis is log(spatial frequency) where the spatialfrequency is measured in 1/mm, and the y-axis is log(P²).

FIG. 6B (bottom row of three graphs) plots the power at one specificspatial frequency over time (in seconds). Again, the left graph is forthe red component, the middle graph is for the green component and theright graph is for the blue component. The selected frequency is shownby the dashed line in FIG. 6A. It is a frequency that exhibits a largedegradation due to the fogging. The strength of the spatial frequencydecreases over time in all three color channels.

In FIGS. 6A-B, the power spectra for each color channel is determined asfollows. The captured lightfield image is cropped to remove null spacein the image. The image has a circular region of interest. This featurewould appear as a high frequency signal in Fourier space. A smoothingwindow applied to the image reduces the impact of the high frequencyfeature. A commonly used smoothing window is the Hann window. This isapplied to the image g and a discrete Fourier transform G is taken. Thepower spectra is recovered from the absolute value of the square of G.Radial samples of G are collected to record the average power spectra.In FIG. 6 above, 400 radial samples of P were collected and plotted on alog-log scale.

Measuring the power spectra of the R, G, and B channels over timemonitors the intensity of spatial frequency contributions in the image.This can be used to control the heating element. This is just oneexample based on the power spectra of the images. In other approaches,control can be based on a wavelet analysis of the images, or on agradient analysis of the images, or on a contrast of the images, or onother measures of the spatial frequency content of the images. In thesecases, the controller may be a microcontroller or embedded processor,rather than a simple feedback loop.

FIGS. 7-8 show the same plots as FIGS. 5-6, but with the active heaterelement turned on. The ear canal phantom was set to 41.7° C. and 97%humidity. A lightfield otoscope was used to produce the images in FIG. 7over a period of 24 seconds. The heating element is active, preventingfog to collect on the front optical surface. The images clearly showclear images over time.

FIG. 8A shows power spectra results for red, green and blue components.The curves for different times are not distinguishable. FIG. 8B plotsthe power at one specific spatial frequency over time, for red, greenand blue components. The selected frequency is shown by the dashed linein FIG. 8A. It is the same frequency as selected in FIG. 6. The strengthof the spatial frequency remains almost constant over time in all threechannels.

Experiments using the ear canal phantom were repeated over a range of36-45° C. and 70-105% humidity, with the heating element on and off. Foreach trial, we observe whether the image remains foggy or clear. A clearimage is a success and is marked by a circle in FIGS. 9A-9B. A foggyimage is a failure and is marked by an x in FIGS. 9A-9B. The results ofthese trials are shown in FIGS. 9A and 9B. FIG. 9A are the trials withthe heating element off. Fogging is apparent starting around 80%humidity. FIG. 9B are the trials with the heating element on. Fogging issignificantly improved.

Although the detailed description contains many specifics, these shouldnot be construed as limiting the scope of the invention but merely asillustrating different examples and aspects of the invention. It shouldbe appreciated that the scope of the invention includes otherembodiments not discussed in detail above. For example, the approachdescribed above is not limited to nichrome wires. Other active heatingelements include metal heating elements, ceramic heating elements,resistive heating elements, thin film heating elements, wire heatingelements. Examples of materials include kanthal and cupronickel inaddition to nichrome.

In addition, this approach can also be used with other imaging systems,including other lightfield systems and also conventional imagingsystems. For example, imaging systems used in body cavities includeendoscopes and otoscopes. Other imaging systems may be designed for usein steam environments, or high temperature or high humidityenvironments. The ability to automatically defog a camera removes theneed for a human to access a camera in a location that is difficult toaccess. Temperature and humidity sensors can be used to automaticallyregulate the temperature of the optics based on the surroundingenvironmental conditions. One example is an outdoor security camera,where weather is constantly changing and condensation can occurnaturally.

Various other modifications, changes and variations which will beapparent to those skilled in the art may be made in the arrangement,operation and details of the method and apparatus of the presentinvention disclosed herein without departing from the spirit and scopeof the invention as defined in the appended claims. Therefore, the scopeof the invention should be determined by the appended claims and theirlegal equivalents.

What is claimed is:
 1. A lightfield otoscope comprising: a housing witha tip configured to receive a disposable speculum; a microlens array anda sensor array; an optical train contained within the housing, theoptical train having a front surface and comprising an objective lensand relay lens, the objective lens positioned at least partially withinthe tip, the relay lens positioned to relay an image plane of theobjective lens to the microlens array and to relay a pupil plane of theobjective lens to the sensor array; and an active heating elementcontrolled by a controller, the active heating element contained withinthe housing and positioned to heat the front surface, thereby reducingfogging and/or condensation on the front surface.
 2. The lightfieldotoscope of claim 1 wherein the controller controls the active heatingelement based on at least one of a temperature and a humidity in avicinity of the front surface.
 3. The lightfield otoscope of claim 1wherein the controller controls the active heating element based on atleast one of a spatial frequency content of images captured by thelightfield otoscope, of a power spectra of images captured by thelightfield otoscope, and a contrast of images captured by the lightfieldotoscope.
 4. The lightfield otoscope of claim 1 wherein the controllercontrols the active heating element based on at least one of a waveletanalysis of images captured by the lightfield otoscope, and a gradientanalysis of images captured by the lightfield otoscope.
 5. Thelightfield otoscope of claim 1 wherein the controller controls theactive heating element based on manual instructions from an operator ofthe otoscope.
 6. The lightfield otoscope of claim 1 further comprising alens barrel that holds the objective lens, the active heating elementheating the lens barrel and the lens barrel conducting heat to the frontsurface.
 7. The lightfield otoscope of claim 1 wherein the activeheating element is a resistive wire having two ends and a mid-section,the two ends located in a base of the tip and connected to thecontroller, the mid-section located near the front surface, and the wirerunning from the base to the front surface and back.
 8. The lightfieldotoscope of claim 1 wherein the controller controls the active heatingelement to avoid injury to a human ear canal.
 9. The lightfield otoscopeof claim 1 wherein the controller has an operating range that includes36-40 degrees Celsius.
 10. The lightfield otoscope of claim 1 furthercomprising: an insulating layer positioned to shield an externalenvironment from heating by the active heating element.
 11. Thelightfield otoscope of claim 1 wherein the active heating elementincludes at least one of a metal heating element and a ceramic heatingelement.
 12. The lightfield otoscope of claim 1 wherein the activeheating element includes at least one of a resistive heating element, aresistive wire and a thin film heating element.
 13. The lightfieldotoscope of claim 1 wherein the active heating element comprises atleast one of nichrome, kanthal and cupronickel.
 14. The lightfieldotoscope of claim 1 wherein the controller comprises an electroniccircuit.
 15. The lightfield otoscope of claim 1 wherein the controllercomprises a microcontroller.
 16. A digital imaging system comprising: anoptical train that forms an image of an object, the optical train havinga front surface; a sensor array that captures the image of the object;and an active heating element controlled by a controller, the activeheating element mounted with the optical train and positioned to heatthe front surface, thereby reducing fogging and/or condensation on thefront surface.
 17. The digital imaging system of claim 16 wherein thedigital imaging system is a lightfield digital imaging system.
 18. Thedigital imaging system of claim 16 wherein the digital imaging system isa digital otoscope.
 19. The digital imaging system of claim 16 whereinthe digital imaging system is a digital endoscope.
 20. The digitalimaging system of claim 16 wherein the digital imaging system is adaptedto be used in a steam environment, a high temperature environment or ahigh humidity environment.